We present a mixed reality-based underwater robot arm teleoperation system with a reaction torque indicator via bilateral control (MR-UBi). The reaction torque indicator (RTI) overlays a color and length-coded torque bar in the MR-HMD, enabling seamless integration of visual and haptic feedback during underwater robot arm teleoperation. User studies with sixteen participants compared MR-UBi against a bilateral-control baseline. MR-UBi significantly improved grasping-torque control accuracy, increasing the time within the optimal torque range and reducing both low and high grasping torque range during lift and pick-and-place tasks with objects of different stiffness. Subjective evaluations further showed higher usability (SUS) and lower workload (NASA--TLX). Overall, the results confirm that \textit{MR-UBi} enables more stable, accurate, and user-friendly underwater robot-arm teleoperation through the integration of visual and haptic feedback. For additional material, please check: https://mertcookimg.github.io/mr-ubi

Four-channel bilateral control is a method for achieving remote control with force feedback and adjustment operability by synchronizing the positions and forces of two manipulators. It is expected to significantly improve the operability of remote control for contact-rich tasks, and in recent years, it has also been used as a data collection method in imitation learning. Among these, the 4-channel bilateral control on the Cartesian coordinate system is advantageous in that it can be used for manipulators with different structures and that the dynamics in the Cartesian coordinate system can be adjusted by adjusting the control parameters, thus achieving intuitive operability for humans. However, achieving high operability by controlling a Cartesian coordinate system remains challenging. In the case of joint space control, all complex interactions between joints are treated as unknown disturbances, and a certain degree of control can be achieved by combining a linear control system with a classical single-input single-output (SISO) system. However, when designing a control system in the Cartesian coordinate system, the position and posture of the manipulator's end-effector are expressed in a three-dimensional special Euclidean group (SE(3)), which has different properties from the vector spaces commonly used in traditional control methods, such as noncommutativity and the fact that addition is not defined. Therefore, it is not possible to use classical control design methods that assume vector spaces as they are. It is possible to approximate the vector space and perform control based on the assumption that the posi-a) Correspondence to: yamane.koki.td@alumni.tsukuba.ac.jp

In recent years, the advancement of imitation learning has led to increased interest in teleoperating low-cost manipulators to collect demonstration data. However, most existing systems rely on unilateral control, which only transmits target position values. While this approach is easy to implement and suitable for slow, non-contact tasks, it struggles with fast or contact-rich operations due to the absence of force feedback. This work demonstrates that fast teleoperation with force feedback is feasible even with force-sensorless, low-cost manipulators by leveraging 4-channel bilateral control. Based on accurately identified manipulator dynamics, our method integrates nonlinear terms compensation, velocity and external force estimation, and variable gain corresponding to inertial variation. Furthermore, using data collected by 4-channel bilateral control, we show that incorporating force information into both the input and output of learned policies improves performance in imitation learning. These results highlight the practical effectiveness of our system for high-fidelity teleoperation and data collection on affordable hardware.

Autonomous manipulation in everyday tasks requires flexible action generation to handle complex, diverse real-world environments, such as objects with varying hardness and softness. Imitation Learning (IL) enables robots to learn complex tasks from expert demonstrations. However, a lot of existing methods rely on position/unilateral control, leaving challenges in tasks that require force information/control, like carefully grasping fragile or varying-hardness objects. As the need for diverse controls increases, there are demand for low-cost bimanual robots that consider various motor inputs. To address these challenges, we introduce Bilateral Control-Based Imitation Learning via Action Chunking with Transformers(Bi-ACT) and"A" "L"ow-cost "P"hysical "Ha"rdware Considering Diverse Motor Control Modes for Research in Everyday Bimanual Robotic Manipulation (ALPHA-$\alpha$). Bi-ACT leverages bilateral control to utilize both position and force information, enhancing the robot's adaptability to object characteristics such as hardness, shape, and weight. The concept of ALPHA-$\alpha$ is affordability, ease of use, repairability, ease of assembly, and diverse control modes (position, velocity, torque), allowing researchers/developers to freely build control systems using ALPHA-$\alpha$. In our experiments, we conducted a detailed analysis of Bi-ACT in unimanual manipulation tasks, confirming its superior performance and adaptability compared to Bi-ACT without force control. Based on these results, we applied Bi-ACT to bimanual manipulation tasks. Experimental results demonstrated high success rates in coordinated bimanual operations across multiple tasks. The effectiveness of the Bi-ACT and ALPHA-$\alpha$ can be seen through comprehensive real-world experiments. Video available at: https://mertcookimg.github.io/alpha-biact/

This paper delves into various robotic manipulation control methods designed for dynamic contact tooling operations on a robotic repair platform. The explored control strategies include hybrid position-force control, admittance control, bilateral telerobotic control, virtual fixture, and shared control. Each approach is elucidated and assessed in terms of its applicability and effectiveness for handling contact tooling tasks in real-world repair scenarios. The hybrid position-force controller is highlighted for its proficiency in executing precise force-required tasks, but it demands contingent on an accurate model of the environment and structured, static environment. In contrast, for unstructured environments, bilateral teleoperation control is investigated, revealing that the compliance with the remote robot controller is crucial for stable contact, albeit at the expense of reduced motion tracking performance. Moreover, advanced controllers for tooling manipulation tasks, such as virtual fixture and shared control approaches, are investigated for their potential applications.

Autonomous manipulation in robot arms is a complex and evolving field of study in robotics. This paper introduces an innovative approach to this challenge by focusing on imitation learning (IL). Unlike traditional imitation methods, our approach uses IL based on bilateral control, allowing for more precise and adaptable robot movements. The conventional IL based on bilateral control method have relied on Long Short-Term Memory (LSTM) networks. In this paper, we present the IL for robot using position and torque information based on Bilateral control with Transformer (ILBiT). This proposed method employs the Transformer model, known for its robust performance in handling diverse datasets and its capability to surpass LSTM's limitations, especially in tasks requiring detailed force adjustments. A standout feature of ILBiT is its high-frequency operation at 100 Hz, which significantly improves the system's adaptability and response to varying environments and objects of different hardness levels. The effectiveness of the Transformer-based ILBiT method can be seen through comprehensive real-world experiments.

Object grasping is an important ability required for various robot tasks. In particular, tasks that require precise force adjustments during operation, such as grasping an unknown object or using a grasped tool, are difficult for humans to program in advance. Recently, AI-based algorithms that can imitate human force skills have been actively explored as a solution. In particular, bilateral control-based imitation learning achieves human-level motion speeds with environmental adaptability, only requiring human demonstration and without programming. However, owing to hardware limitations, its grasping performance remains limited, and tasks that involves grasping various objects are yet to be achieved. Here, we developed a cross-structure hand to grasp various objects. We experimentally demonstrated that the integration of bilateral control-based imitation learning and the cross-structure hand is effective for grasping various objects and harnessing tools.

You know, that thing where the flow suddenly slows to a halt and you inch forward for a half hour and then things pick up again and you look around for an accident or construction or anything at all for Pete's sake that might justify the time you just wasted. It's as if the fates chose this particular time and place to screw with you. That may seem counterintuitive, since you don't have much control over how far you are from the car behind you--especially when that person is a tailgater. But the math says that if everyone kept an equal distance between the cars ahead and behind, all spaced out in a more orderly fashion, traffic would move almost twice as quickly. Now sure, you're probably not going to convince everyone on the road to do that.